Partial local entropy and anisotropy in deep weight spaces

We refine a recently-proposed class of local entropic loss functions by restricting the smoothening regularization to only subset weights. The new are referred as partial entropies. They can adapt weight-space anisotropy, thus outperforming their isotropic counterparts. support theoretical analysis with experiments on image classification tasks performed multi-layer, fully-connected and convolutional neural networks. present study suggests how better exploit anisotropic nature deep landscapes provides direct probes shape minima encountered stochastic gradient descent algorithms. As by-product, we observe an asymptotic dynamical regime at late training times where temperature all layers obeys common cooling behavior.